CI-260 - Transfer dataset files concurrently - #224

Purpose

Speed up dataset upload and download. Transfer time previously scaled with the number of files rather than total bytes, because every file paid a large fixed cost and files were transferred strictly one at a time.

How

1. Transfer files concurrently. upload_directory and download_directory now resolve the work list, then run several transfers at once through a ThreadPoolExecutor, reporting into one aggregate progress bar instead of one bar per file.
2. Reuse the transfer manager. upload_fileobj/download_file each built and tore down a boto3 TransferManager (~16 threads) per file. S3Client now holds one for its lifetime and shuts it down when the batch ends. Local files are uploaded by filename so s3transfer reads their parts in parallel; Path-like objects that are not local files (see Change S3 Client upload implementation to use upload_fileobj #115) are streamed through their own open() into the same shared manager.
3. Stop rebuilding the S3 client per file. DataPortalFiles.download called download_files once per file, constructing a fresh boto3.Session each time (~300 ms measured). It now makes a single batched call. File objects are also passed through instead of being flattened to strings, which drops one HeadObject per file.
4. --threads to control it. Available on upload, download and resume-upload, and as a threads parameter on the service and SDK methods. Default 8. --threads 1 restores fully sequential behaviour: no ThreadPoolExecutor, and TransferConfig(use_threads=False) so boto3 runs the transfer inline too — which also makes this usable where threads are unavailable, such as Pyodide.

Behaviour changes

• An upload that exhausts its retries now raises, naming every failed file. It previously returned normally, so a partially-uploaded dataset looked complete.
• Retry backoff is capped at 60s (was uniform(0,60) + retry*60, up to ~45 min and blocking every remaining file).
• Two source files mapping to the same destination path now raise instead of racing.

Biggest risk

Concurrency is the new default, so upgrading changes behaviour for everyone. 8 files in flight could surface throttling or contention on networks and tenants unlike the one tested here, and failures that were previously serial and deterministic are now interleaved. Mitigation is --threads N, with --threads 1 restoring the old sequential behaviour.

The risk I expected and ruled out: switching upload_fileobj → upload_file could have changed checksum semantics. It does not (verified below).

Test dataset

301 files, 11.17 MB, uploaded to dev.cirro.bio / Pipeline Development as data type Files (custom_dataset):

• 300 x 4 KB random-byte files across 5 subdirectories — the many-small-files case where per-file overhead dominates
• 1 x 10 MB file, crossing boto3's 8 MB multipart threshold

Both arms ran on the same venv and interpreter (Python 3.14.6, boto3 1.41.5, s3transfer 0.15.0), with PYTHONPATH selecting which cirro loaded, so the only variable is this diff. Released 1.12.1's file_utils.py, clients/s3.py, sdk/file.py and services/file.py are byte-identical to this branch's merge-base, so the baseline is exactly "this branch minus these changes". 2 reps each, interleaved.

Results

The SDK download path gains most because it also carried the per-file client rebuild: subtracting actual transfer time leaves ~90s of pure overhead on the baseline, or ~296 ms per file.

Correctness

• Checksums unchanged. Uploading identical bytes with both versions produced the same ChecksumType=FULL_OBJECT and identical CRC64NVME values, for both the 4 KB file and the 10 MB multipart-threshold file.
• Round trip clean. 301/301 files, 0 size mismatches, 0 CRC mismatches, 11,714,560 bytes.
• validate_folder: 301 matching, 0 not-matching, 0 missing, 0 errors.
• Peak threads plateaued at 24-25 (vs 7-8 baseline), matching 8 workers + 10 request + 5 submission + 1 io, and returned to baseline after each batch.
• Zero Connection pool is full warnings across all 12 runs.
• --threads verified end to end. threads=1 created zero worker threads and round-tripped 12/12 files; threads=8 created 8 and round-tripped 12/12. Neither left threads behind.

All benchmark datasets were deleted and the project reports none remaining.

Note on --threads 1

It removes every thread the transfer logic creates, but tqdm spawns its own monitor thread on the first progress bar. That is pre-existing (the released version does it too) and not affected by this flag.

🤖 Generated with Claude Code